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Abstract: 

With the widespread use of learning analytics (LA) tools, there is a need to explore how these 
technologies can be used to enhance teaching and learning. Little research has been conducted on 
what human processes are necessary to facilitate meaningful adoption of LA. The research 
problem is that there is a lack of evidence-based guidance on how instructors can effectively 
implement LA to support students. The goal of the study was to develop and validate a model to 
guide instructors in the implementation of LA tools. Using design and development research 
methods, an implementation model was constructed and validated internally. Themes emerged 
falling into the categories of adoption and caution with six themes falling under adoption 
including: 

• LA as evidence, 

• reaching out, 

• frequency, 

• early identification/intervention, 

• self-reflection, and 

• align LA with pedagogical intent. 

Three themes emerged falling under the category of caution including: 

• skepticism, 

• fear of overdependence, and 

• question of usefulness. 

The model should enhance instructors’ use of LA by enabling them to better take advantage of 
available technologies to support teaching and learning in online and blended learning 
environments. Researchers can further validate the model by studying its usability (i.e., usefulness, 
effectiveness, efficiency, and learnability), as well as, how instructors’ use of this model to 
implement LA in their courses affects retention, persistence, and performance. 
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Introduction 

Background 

Learning analytics (LA) is the collection, analysis, and reporting of available data to 
improve the teaching and learning process and environment (Siemens & Long, 2011). There are 
two main categories of research in the field of LA. The first is on how to capture, process, and 
present data to educational stakeholders in useful ways. The second, and less common, focus of 
research is on how to take up and use analytics in practice to inform choices or prompt action 
(Wise, Vytasek, Hausknecht, & Zhao, 2016). More simply, the majority of research has focused 
on how to create useful information from large quantities of collected data (Dawson, Gasevic, 
Siemens, & Joksimovic, 2014). Less research has been conducted on how to actually put this 
infonnation to use to achieve desired purposes in the educational environment (Ferguson et ah, 
2014; Lockyer, Heathcote, & Dawson, 2013; West, Heath, & Huijser, 2016; Wise, 2014; Wise et 
ah, 2016). LA holds potential application for a range of stakeholders in higher education including 
instructors, researchers, curriculum developers, learning environment designers, and university 
policy makers. LA is utilized at many levels within academic institutions, but a common 
application is at the course level (Dziuban, Moskal, Cavanagh, & Watts, 2012). Data within the 
learning management system (LMS) regarding student activity can be tracked and analyzed to 
monitor student progress, predict student success or failure, or inform instructional design. LA at 
the course level is an important area of research that promises to improve learning outcomes in 
online and blended courses by providing rich infonnation regarding participation and performance 
to instructors and students alike. 

Much of the literature in the second category of LA research uses the term “intervention” 
to describe the act of taking up and using analytics in practice (Lockyer et al., 2013; Wise, 2014; 
Zacharis, 2015). Wise et al. (2016) pointed out that this term can be useful, but can also include 
the undesired connotation that LA use is an interruption in the regular teaching and learning 
process. Instead, they chose to use the term “LA implementation” to describe the use of LA as an 
ongoing part of the regular monitoring and responsive adjustment to teaching and learning 
practices. This study also uses the term “LA implementation” to describe the process of taking up 
and using analytics in practice. 

Problem Statement 

While LA tools may show that students who regularly log into an LMS perfonn better than 
their less active peers, this infonnation alone changes nothing and does not mean the instructor 
will provide a suitable response (Roll & Winne, 2015). Furthennore, simply telling the student to 
log into the LMS more often will not be helpful (Dawson et al., 2014). While analytics tools may 
provide insight, they do not help instructors to provide a systematic and integrated response to 
such situations that will result in better outcomes for the at-risk student. As Wise (2014) stated, 
“without a plan for shifting patterns of teaching and learning activity, new technologies often 
remain ancillary to the teaching and learning process, either used tangentially to marginally 
enhance existing practices or often simply collecting dust on the virtual shelf’ (p. 203). Little 
research has been done on what human processes are necessary to facilitate meaningful 
implementation of LA. 

There is a general lack of research-based guidance on how various stakeholders (i.e., 
learners, instructors, and administrators) can effectively use LA tools, but researchers have begun 
to address this in recent years. West et al. (2016) presented a framework for institutional 
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implementation of LA to support student retention efforts. Wise et al. (2016) addressed the 
problem of how students can take up and use LA in practice, but many LA tools are designed for 
instructor use and students cannot access the infonnation they generate. Mor, Ferguson, and 
Wasson (2015) focused on how instructors can use LA to inform their reflective practice and 
learning design, but very few studies have focused on how instructors can use analytics in practice 
to support the student learning process. There is a need for a model to support instructor-specific 
use of LA to encourage its systematic use as an integrated part of the teaching process. The research 
problem is that there is a lack of evidence-based guidance on how instructors can effectively 
implement LA in their courses. 

This study focused on the use of LA at Southwestern Oklahoma State University 
(SWOSU). SWOSU is a regional university in western Oklahoma with approximately 5,000 
undergraduate, graduate, and professional students enrolled and approximately 225 faculty 
members employed. SWOSU currently provides faculty with two LA tool options. All faculty have 
access to Canvas Analytics as part of the Canvas LMS. SWOSU is also piloting AspirEdu’s 
Dropout Detective in two of its fully online programs including RN to BSN and Health Information 
Management (HIM). Both of these tools are designed for instructor use. This paper will present a 
brief review of literature concerning LA tools, models, and implementation. The methodology of 
the study will then be presented. Last, the resulting LA implementation model will be presented 
along with conclusions and recommendations for research and practice. 


Literature Review 

A review of the literature guided the identification of what LA tools and models are 
currently available to instructors, how they are being used, and the benefits and limitations of such 
tools and models. This review informed the design and development of a preliminary model to 
guide instructor use of LA. The following review of literature includes a brief overview of the 
current state of the body of knowledge in the LA field regarding data capture, processing, and 
display as well as LA implementation. 

Learning Analytics Tools 

Performance and tracking. The majority of research in the LA field has been on the 
development and validation of LA tools to support student performance tracking. Spivey and 
McMillan (2013) as well as Mo and Zhao (2012) presented research studies focused on using 
Blackboard LMS to track student data. Spivey and McMillan (2013) investigated the relationship 
between student effort and performance by utilizing data already being tracked in Blackboard. The 
researchers found that more frequent access and a more evenly spaced study schedule (as opposed 
to “cramming”) had a positive effect on student perfonnance. Mo and Zhao (2012) had very similar 
findings. Both studies focused on using the tools already built into the LMS to track student data 
to monitor students and analyze effort and performance. Similarly, You (2015) found a link 
between academic procrastination and course achievement when examining LMS data. These 
studies are examples of using the tools at hand to begin implementing the principles of LA in the 
online classroom. 

Mazza and Dimitrova (2007) developed and analyzed a student monitoring tool for 
supporting instructors in online courses. The researchers surveyed users regarding the 
effectiveness, efficiency, and usefulness of their tool and found that the use of graphical 
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representations of data was important to the user. Similarly, Ruiperez-Valiente, Munoz-Merino, 
Leony, and Delgado Kloos (2015) presented a study of another LA tool that visualized data for the 
user. Ali, Hatala, Gasevic, and Jovanovic (2012) presented two evaluations of their tool, LOCO- 
Analyst, which also focuses on visualizing LMS data for instructors, and, last, Macfadyen and 
Dawson (2010) discussed the development and implementation of another dashboard-like tool that 
also visualizes LMS data and found that meaningful information can be extracted from LMS data 
and tools can be developed which visualize student progress and the likelihood of their success. 
All four of these studies concluded that the visualization aspect is important so instructors are able 
to readily discern outliers and points of concern and react to such circumstances quickly. 

Student retention. Another common theme found in the literature on LA tools is the 
development of tools aimed at increasing student retention. Retention efforts begin in the 
classroom, so this topic has many stakeholders and touches every level of higher education. 
Agnihotri and Ott (2014) presented the development of an LA tool aimed at student retention. 
They viewed this issue from an administrative level and sought to provide a tool for retention 
counselors within the university. The purpose of this tool was to provide retention risk ratings to 
counseling staff for each new freshman before the start of the fall semester. Agnihotri and Ott 
(2014) concluded that such tools are capable of increasing student retention, but that the 
development process must utilize a broad perspective of the entire retention process. 

Similarly, Harrison et al. (2015) presented an early alert system designed to identify 
students at risk of discontinuing enrollment. They included demographic, institution, and learning 
environment variables in their model resulting in a tool that could accurately predict those at risk 
of discontinuing. Jayaprakash and Lauria (2014) presented yet another early alert system designed 
to identify students at academic risk for the purpose of increasing student retention rates. Knight 
and Shum (2014) took the discussion of tool development a step further by introducing the idea 
that the design LA tools should be informed by epistemology, assessment, and pedagogy. They 
made the point that it is not the tool itself, but the way in which it is wielded, which determines its 
value. This idea leads to the discussion of LA models to guide the implementation and use of LA 
tools. 

Learning Analytics Models 

More recent research has gone beyond tool development and validation and begun to take 
a broader view of the issue of LA model development and validation. Martinez-Maldonado et al. 
(2015) as well as Scheffel, Drachsler, Stoyanov, and Specht (2014) presented frameworks to 
support the development and evaluation of LA tools, respectively. Ali, Asadi, Gasevic, Jovanovic, 
and Hatala (2013) sought to identify what specific factors would lead instructors to use or not use 
LA tools. Macfadyen and Dawson (2012) pointed out that LA should be consulted and integrated 
into the institutional strategic planning process. Ferguson et al. (2014) presented a framework to 
support the implementation of LA at the institutional level. Although no specific framework or 
model was presented, Dringus (2012) described a number of principles for the adoption of LA 
tools while expressing an attitude of caution when considering LA as being potentially “harmful.” 
Last, West et al. (2016) presented a framework for LA implementation in relation to student 
retention. This framework was meant to stimulate a discussion about the institutional 
implementation of LA. 
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Perhaps most relevant to this study are the frameworks presented by Wise (2014) and Wise 
et al. (2016). Wise (2014) presented a discussion of designing interventions based on the output of 
LA tools pointing out that this part of the process is often ignored and is a relatively unexplored 
area of research. There are three specific aspects of the application of LA: what traces of learning 
should be captured, how to present these traces to learners, and how to frame the inclusion of 
analytics as part of the course activity to guide their use in productive decision-making by learners 
and teachers (Wise, Zhao, & Hausknecht, 2014). These interventions have to do with the latter two 
aspects. Wise (2014) pointed out that as LA tools are becoming more prevalent, intervention 
design becomes critical to their effective implementation and offered the following important 
research questions: when in the teaching and learning process should analytics be consulted; who 
should be accessing analytics; why are they being consulted; and most importantly, how the use 
of the analytics articulates with the rest of the teaching and learning practices taking place. 

Wise (2014) presented a framework which began to answer some of these questions, but a 
revised and extended version of this framework was presented by Wise et al. (2016). They 
presented a model for student use of LA as a part of a self-regulatory cycle of grounding, goal¬ 
setting, action, and reflection, the Student Tuning Model. The Student Tuning Model suggests that 
students engage in a continual cycle of planning, monitoring, and adjusting their learning practices 
as they are informed by analytics. The element of Grounding has to do with the relationship 
between the information the analytics provide and the specific educational context in which they 
are being provided. Students must understand the purpose of the learning activity, what represents 
meaningful engagement in the activity, and how the LA provided will reflect this to the student. 
Goal-Setting has to do with the student planning specific objectives and actions for reaching them 
in relation to the larger context established through Grounding. Action is when students engage in 
behaviors to realize their goals. Reflection occurs when students use analytics to reflect on the 
actions they took in comparison to the goals they set. 

The Student Tuning Model was meant to outline how students might productively engage 
with analytics. Wise et al. (2016) also provided a framework for pedagogical design to support 
student use of analytics. The Align Design Framework includes the four principles of Integration, 
Agency, Reference Frame, and Dialogue/Audience. The first principle of Integration states that 
the instructor should position student analytics use as an integral part of the learning process. The 
second principle of the framework is Agency which has to do with students taking ownership of 
their learning process. The principle of Reference Frame states that instructors should provide a 
comparison point to students. The final principle of this framework is Dialogue/Audience. This 
principle states that the instructor should create an environment where interpretation of analytics 
is discussed between the instructor and students so that students don’t simply feel that they are 
being watched. 

While this framework is a good starting point, the research problem remains that there is a 
lack of evidence-based guidance on how instructors can effectively implement LA tools which are 
designed to present information only to the instructor and not the student. Wise’s (2016) 
framework does little to help in this situation. Lockyer et al. (2013) addressed this issue in part by 
presenting the idea that a conceptual framework should be established for typical LA patterns 
expected from particular learning designs in order to better help teachers interpret the information 
that analytics provides. Lockyer’s model has a narrow focus on how learning design can inform 
the use of LA and is difficult to generalize to a variety of learning situations. 
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Method 

A qualitative design and development research approach (Richey & Klein, 2007) was used 
to address the research problem that there is a lack of evidence-based guidance on how instructors 
can effectively implement learning analytics (LA). Specifically, model construction and validation 
methods were used to construct an instructor LA implementation model. The study took place 
within SWOSU and focused on the use of the LA tools available there. 

First, the review of literature served as the basis for answering the first research questions: 
(a) what LA tools and models are currently available to instructors, (b) how are they using these 
tools and models to support teaching and learning, and (c) what are the benefits and limitations of 
such LA tools and models? Next, a needs assessment was conducted to address the second research 
question: what needs to be considered to design an effective model to guide instructors in using 
LA tools and implementing interventions? A survey and a follow-up focus group were used to 
identify needs of stakeholders including instructors, online learning administrators, and online 
learning committee members. Then, a preliminary model to guide instructors in the use of LA tools 
was designed based on the review of literature and the needs assessment which addressed the third 
research question: how can stakeholder needs inform the design of such a model? The next phase 
included an expert review of the model using Delphi panel technique. This approach addressed the 
fourth research question: how do instructors perceive the effectiveness and efficiency of the 
proposed LA model? Last, modifications were made to the model to implement suggestions from 
the Delphi panel, which addressed the fifth research question: what modifications are needed to 
improve the proposed LA model? This three-phase process (i.e., needs assessment, model 
construction, and model validation) resulted in a model, which is useful to instructors wanting to 
effectively implement LA tools in their courses. 


Results 

Overview 

This study was designed to identify stakeholder needs regarding the implementation of LA 
at the course level in order to develop and validate a model to support instructor use of LA. The 
researcher began by conducting a needs assessment including a survey and two focus group 
sessions. The survey was used to collect quantitative and qualitative data from instructors 
regarding the use of LA in their courses. The survey included questions in the categories of 
demographics, prior use and perceptions, efficacy, model construction, and focus group 
participation. The focus group sessions were meant to elicit more detailed information from 
participants. The first session had seven participants in attendance, and the second had ten. Next, 
data from the survey and focus group sessions were analyzed in the context of the research 
questions and a model was developed based on the review of literature and analysis of the data. 
Last, the model was reviewed by a Delphi panel until consensus was reached. The model was 
approved by the panel, which serves as internal validation. 

Model Construction and Validation 

The survey was sent to approximately 350 full-time and adjunct faculty from both SWOSU 
campuses. There were 61 (i.e., 17.42%) responses to the survey. The low response rate is explained 
by the fact that the survey was sent to both SWOSU campuses as well as adjunct instructors. Many 
of the recipients were unfamiliar with the topic of LA and uninterested in the study. The results of 
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the survey items in the categories of prior use and perceptions and efficacy confirmed the desire 
to implement LA in the classroom and the need for instruction on how to effectively do so. The 
next section of the survey asked the open-ended question: “What additional information or training 
would enable you to use learning analytic tools (e.g., Canvas Analytics or Dropout Detective) more 
effectively?” Of the 61 respondents, 34 provided a response to this question. These qualitative data 
was analyzed along with the focus group data. The next section of the survey asked participants if 
they would be willing to participate in a focus group. Of the 49 participants who answered this 
question, 31 responded that they would be willing to participate (63.3%). Those who answered yes 
provided their contact information (name, email address, phone number) in the final section of the 
survey. 

The next phase of the study were the focus groups. The researcher conducted two focus 
group sessions because of the large number of willing participants. There were initially 31 survey 
respondents who stated they would be willing to participate. Once scheduled, the focus group 
sessions were attended by a total of 17 participants. A semi-structured approach was used, with 
the researcher using the focus group protocol to loosely guide the discussion and asking follow up 
questions when necessary. The researcher took brief notes and had a teaching assistant take an 
additional set of notes. Each focus group session was scheduled to run about one hour. Both 
sessions ran about 15 minutes over the allotted hour due to rich discussion that occurred. The notes 
from the focus group sessions, as well as the text from the qualitative survey item, were loaded 
into qualitative research software which was used to analyze, identify themes, and code the data 
according to the themes. 

The resulting model, which is included in the next section, was validated internally using 
a Delphi panel method. The researcher recruited three participants from the focus group sessions 
to participate in the Delphi panel. These three participants were considered subject matter experts 
in LA at SWOSU. Their participation in the focus group sessions also enabled them to assess 
whether the model addressed the needs and opinions voiced during the focus group session. The 
model was sent to the panel by email, and they were asked to complete a questionnaire to assess 
whether the model adhered to what was discussed during the focus group as well as the usability 
of the model according to the Rubin and Chisnell’s (2008) attributes. For the most part, the Delphi 
panel found the model to be complete, useful, efficient, effective, and learnable. Three suggestions 
for improvement were made, but after discussion and clarification of the model's purpose, it was 
determined that the suggestions were beyond the scope of the model’s intent and the model was 
approved. This served as internal validation of the following model. 

Resulting Model 

Based on a review of the current literature regarding LA and a needs assessment (including 
a survey and two focus group sessions) regarding LA implementation at SWOSU, the following 
instructor LA implementation model was developed (Figure 1). The first focus group session was 
very positive and implementation strategies were discussed and refined. The second group 
expressed a very cautious attitude toward the implementation of LA. It became clear during the 
focus group sessions that the themes identified fell into two broad categories: adoption and caution. 
These contrasting attitudes reflect the various tones of literature concerning LA implementation 
(Dringus, 2012; Wise, 2014). Although themes fell into these two seemingly conflicting groups, 
the model is meant to demonstrate that both adoption and caution are part of the overall 
implementation process. Themes are organized according to these two categories, and practical 
and conceptual guidelines are presented based on these themes. 
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Figure 1. Instructor Learning Analytics Implementation Model 


Adoption. Many instructors participating in the focus groups already used LA extensively 
in their courses. Others were eager to leam more and begin the implementation process. Based on 
the needs assessment and review of literature the following themes emerged: LA as evidence, 
reaching out, frequency, early identification/intervention, self-reflection, and aligning LA with 
pedagogical intent. These themes fell under the category of Adoption. 

LA as Evidence. It seems that many instructors appreciate that LA provides indisputable 
facts and information. This type of information can be used in a variety of ways, but it is a common 
theme that instructors appreciate the ability to look up and report hard data. For example, this 
information can be used to confirm or dispute a student’s story if he claims computer issues 
prevented him from completing his work. It might also help to support an instructor if a grade is 
disputed because LA can track student activity as well as student/instructor communication. 
Instructors might also use charts, graphs, etc. from an LA tool when reaching out to a struggling 
student. These data can help justify instructor concern and persuade the student that there is a 
problem that needs to be addressed. Instructors might also benefit from their students knowing that 
this information is readily available. If a student knows that the instructor can see a high level of 
detail on student course activity, this increases accountability on the student’s part. He will feel 
that his actions matter and someone is paying attention. 

It can be very difficult for instructors to remember details regarding student activity, 
communication, etc. When implementing LA in a course, instructors should remember that these 
tools are there to support their teaching practice. When questions arise, instructors should 
remember to consult these tools because they often reveal more information than instructors can 
readily recall themselves. In addition, when contacting students regarding participation, activity, 
or grades, it might be helpful to include data generated by LA in that line of communication. This 
evidence helps students understand that instructors are not relying solely on instincts or memory, 
but that specific facts and details are available. 

Reaching out. Many instructors expressed that LA helps them to reach out to students who 
are struggling and can result in a better relationship. Instructors often use the information generated 
by LA to identify students who are struggling or falling behind, and “reach out” to these students 
by contacting them personally. This simple act is often enough to help students improve because 
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it lets them know that someone notices and cares. One focus group participant said it “shows the 
students that you are aware of what they are doing and how they are perfonning. Giving them a 
heads up early on makes them feel that they are in charge of their performance. Sometimes they 
may just admit that they are lazy, but at least it is up to them how they will move forward. It also 
lets them know that you can identify problems.” Another said that acting on non-participation lets 
students know they are missed. 

Instructors can use LA tools to identify students who are struggling and initiate some kind 
of conversation with them. Sometimes students might just need a little nudge. Often students in 
large or online courses feel that no one notices whether they succeed or fail, and even a few words 
can make a big difference. Instructors teaching large or online courses know that it is difficult to 
monitor the progress of so many students when instructors often do not ever meet these students 
face-to-face. The job is not easy. LA tools can make that job a bit easier so instructors can be more 
effective in reaching out. 

Frequency. A useful strategy is to consult LA tools consistently as the course progresses. 
Many instructors make a habit of consulting these tools once or twice per week to see if there is 
any new information to act upon. This consultation provides instructors with information on 
student activity in addition to what is observable from the course itself. How often these tools 
should be consulted depends on the course structure. What is important is to develop a schedule 
that works for the course and abide by it. Wise et al. (2016) stated “the frequency with which the 
analytics are provided or accessed as well as the schedule for reflective activity will vary 
depending on the context. The goal is to create a specific timing for cyclical review” (p. 12). 

Early identification/intervention. Many instructors feel that LA tools are most beneficial 
early in the course because it is important to identify struggling students early when there is still 
time to get them back on track. One focus group participant suggested always having an 
assignment due during the first week of the course and using these tools to see which students are 
not putting that effort in right off the bat. Identifying and intervening early with these students is 
key. Another participant recommended identifying where the “point of no return” is in each course 
and being mindful as it approaches. Helping students get on track with the course before this point 
can increase the probability of success. 

Additionally, it is important for instructors to develop consistent intervention strategies to 
use when acting upon the information provided by LA tools. Many instructors benefit from the use 
of prefonnatted messages. These messages can be used to reach out to struggling students, advise 
them on where to find help, and direct them to campus resources such as retention, tutoring, writing 
center, etc. These messages should by no means be restricting and should be edited and customized 
to whatever degree the instructor prefers, but having preformatted messages makes this kind of 
communication more consistent and can save instructors’ valuable time. It is also beneficial to 
decide beforehand what constitutes a need for intervention and what kind of intervention is 
appropriate. Many instructors develop a flowchart or similar visual depiction of their policies, 
which helps them to decide when and how to intervene. A flowchart like this also provides 
consistency and saves time. 

Self-reflection. A common theme in the literature as well as in the needs assessment is the 
use of LA for the purpose of self-re flection. LA can provide a wealth of infonnation to instructors 
wanting to assess their course and teaching practices. Focus group participants discussed how LA 
can be used to analyze teaching and adjust courses based on findings (e.g., when students are 
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actively involved, when they lack interest, and where there are areas of improvement). Using LA 
for test item analysis is useful to this end. Analyzing which exam questions are most frequently 
missed can reveal what teaching areas need more focus or perhaps might reveal some “bad 
questions.” One focus group participant mentioned that she uses LA to see what level of instructor 
discussion participation results in higher student evaluations. This helps her to identify how much 
participation is appropriate so as not to monopolize the conversation or have too small a presence. 

Instructors wanting to implement LA in their course structure can greatly benefit from 
using LA as a tool of self-reflection. The infonnation can supplement the traditional course and 
instructor evaluation and perhaps reveal more detailed information. This type of self-reflective 
activity can take place throughout the teaching and learning process, but also at the end of each 
semester before beginning another. Instructors can use what they learned from LA in one semester 
as they design and make changes to the course for the next semester. 

Align LA with pedagogical intent. The last theme of LA adoption identified in the 
literature and needs assessment is that the use of LA tools must align with the instructor’s 
pedagogical intent. These tools are not one-size-fits-all. There are some circumstances in which 
certain features are not useful in a course. There are even situations where LA is not useful at all 
in a course. Instructors must always be mindful of what is being measured and reported and 
whether this information is an accurate reflection of learning based on their course design. 
Pedagogy must drive the use of LA. Wise et al. (2016) presented the Align Design Framework 
which offered principles for pedagogical practice to support the use of LA. The idea is that 
instructors can adjust their pedagogy to support LA implementation. While there is an important 
relationship between pedagogy, course design, and the use of LA, focus group participants felt that 
pedagogy is of greatest importance and the use of LA must be aligned and adjusted to fit the 
developed pedagogy. One focus group participant stated, “You must analyze what elements of LA 
tools will add value to your course and know this before the course begins so you have a plan for 
how to use LA in your course.” It is important for instructors to understand what is being measured 
by these tools and how, consider how these measures align with the course structure and pedagogy, 
and remember this when consulting these tools and acting on the information they provide. 

Another participant noted that the use of LA also depends on the interest of the faculty 
member. This model is useful for faculty who desire to utilize LA, but the use of these tools should 
not be forced. Some instructors are not interested in these tools and feel that they can serve their 
students and develop relationships without the use of this type of technology. LA should only be 
used to supplement and assist instructors but will never be able to replace the personal connection 
between instructors and students. 

The idea of aligning the use of LA with the instructor’s pedagogical intent was discussed 
from a number of perspectives relating to the implementation and adoption of LA, but it was also 
discussed from a cautionary perspective. Many participants felt that instructors implementing LA 
in their courses must be wary of these tools and consider how much weight should be placed on 
the infonnation they reveal. These concerns relate to the second category of themes, which reflect 
an attitude of caution. 

Caution. Much of the literature, as well as the qualitative data collected in the needs 
assessment, revealed a very cautious attitude towards the implementation of LA in the classroom. 
Many felt that these tools can be inaccurate, impersonal, or intrusive. It is common for users to be 
wary of new technologies, and LA is no exception. A number of themes emerged within this 
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category such as: skepticism, fear of overdependence, and the questioning of the overall usefulness 
of LA. 

Skepticism. If LA is going to be useful in a course, it is essential that the use of LA aligns 
with the instructor’s pedagogical intent; however, many instructors question whether this can be 
the case. When these tools are not transparent about how they collect, analyze, and report data, 
instructors become skeptical as to whether the data can be trusted. The way these tools measure 
student success is not always representative of the students’ effort and performance. In addition, 
different tools use different metrics so it is difficult to compare them. Transparency is essential if 
instructors are going to trust that the information provided by LA tools is accurate and can be acted 
upon. One participant noted that she wants to see exactly what measures are going into the 
algorithms that detect and label “at-risk” students. 

Similarly, there is concern that LA is too often about the bottom line and does not take the 
cultural context of the students and campus into account. An example of this is that many students 
at SWOSU work full time, often on a family farm. These students might begin to struggle to keep 
up, and LA does not reflect these types of situations. LA is unable to identify students who are 
personally at-risk in some way rather than academically at-risk. While LA cannot detect this level 
of detail regarding students’ personal circumstances, they can accurately reflect symptoms of a 
deeper problem. One participant noted that these tools must be used critically to help instructors 
understand these underlying causes. 

Another concern is that students may begin to understand what activity these tools measure 
and how they measure it, and these students may begin to “work the system.” For example, if an 
LA tool measures how long students are logged into the LMS, they may log in and stay logged in 
while working on other things and not actively engaged in the course. Another example is if an 
LA tool measures the number of clicks (e.g., click tracking software) students may use this to their 
advantage by clicking their mouse randomly to increase their participation level. One participant 
said that students might think “you want more clicks, I’ll give you more clicks!” Some LA tools 
measure student perfonnance in relation to the perfonnance of the class as a whole. Some 
participants expressed concern that students may attempt to take advantage similarly to when a 
class is graded on a curve. Everyone underperforms because they know their performance is 
measured as it relates to the class as a whole. While it is uncommon for instructors to actually 
assign grades based on LA data, the concern about this misuse of LA is real. 

Participants expressed the fear that LA tools may encroach on privacy in some way. This 
concern has already been expressed in the literature on LA. Picciano (2014) pointed out that “as 
well-intentioned as LA might be in terms of helping students succeed, this ‘big data’ approach 
may also be seen as ‘big brother is watching’ and, as such, an invasion of privacy that some 
students would find objectionable” (p. 41). Many fear that it might make students uncomfortable 
for instructors to have this level of detailed information, but they also fear that administration will 
use this information to monitor instructor performance. This fear of surveillance is closely related 
to the fear that these tools do not always measure performance accurately because there is no “one- 
size-fits-all.” The concern is that administration will use LA destructively to monitor employees, 
which might create a privacy issue. 

One participant mentioned that faculty and administration alike must come to a level of 
“rhetorical literacy” in order to make proper use of LA. Selber (2004) introduced the idea that 
there are different levels of literacy, which can be developed regarding the use of technology: 
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functional literacy (computers as tools), critical literacy (computers as cultural artifacts), and 
rhetorical literacy (computers as hypertextual media). The participant noted, “The basic idea is 
functional literacy is the most basic kind of usage of technology, while rhetorical literacy requires 
a much more sophisticated self-awareness of the technology user. Selber (2004) argues that most 
users get stuck in the critical literacy stage and think that there is no other place to go, especially 
when it comes to using technologies responsibly and ethically.” This participant felt that users of 
LA tools should reach a level of rhetorical literacy in order to use LA properly, but also felt that is 
unlikely to happen. The main concern was that administration could inappropriately use this 
technology to monitor instructors without having a true understanding of the technology, the 
course, the instructor, or the pedagogy. Rhetorical literacy would mean that these things are 
critically understood which would enable users to make effective use of LA. Many participants 
felt that LA should be used as a tool, not a weapon. 

Fear of overdependence. A similar theme found in the needs assessment is the fear that 
users will become overly dependent on these tools. The concern is that faculty and administration 
might put too much stock into these tools and treat them as the “end-all-be-all” solution to the 
problem of helping at-risk students and increasing retention. One participant noted that it is a 
problem in our society in general for people to want a quick-fix answer or something that will 
make everything better, but that is not how it works. There is a time and a place for LA. He advised 
to not be too critical or too enthusiastic about the use of LA. Just as LA should be used as a tool, 
not a weapon, users need to remember that it is only one tool in the toolbox. 

Question of usefulness. Finally, some instructors question the overall usefulness of LA 
tools. Many mentioned that some students are just not prepared for a course and there are no 
interventions that would enable the student to succeed. One participant also questioned to what 
degree instructors should commit to helping the students succeed, and what should simply be left 
to the student. While it is ultimately up to the student to succeed in a course, instructors should 
also be available and willing to use whatever resources and time they have available to support 
students. LA tools ultimately save instructors time and act as an assistant for instructors wanting 
to look deeper into the level of student participation. 

Model Conclusions. It is important to be mindful of these themes and cautious about the 
implementation of LA, but these concerns do not mean that LA cannot be implemented 
successfully when approached cautiously. Instructors should remember that LA is a powerful tool, 
but should not be used as a weapon, and this tool is only one in the toolbox. LA is not a quick fix 
answer that will ease all of the retention problems faced by instructors, but it can serve to assist 
them in their efforts to support students, which is the ultimate goal. These tools must be used 
critically while seeking to reach a level of rhetorical literacy concerning this new technology, 
which can greatly benefit students and instructor practice if implemented appropriately and 
effectively. 
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Discussion 

The purpose was to develop and validate an instructor learning analytics (LA) 
implementation model. This model was developed to enable instructors to effectively implement 
whatever LA tools they have available in their courses. Although based on the research conducted 
at a single institution using only two available LA tools, the model is intended to be generalizable 
to a number of environments and LA tools. A thorough review of the existing literature on LA as 
well as a needs assessment guided the development of the model. 

Implications 

This study helped to identify the needs of instructors wanting to implement LA in their 
courses. The results informed the design of an instructor LA implementation model. The model 
was validated internally by a panel of experts. The final model includes practical and conceptual 
guidelines regarding the use of LA and is meant to be generalizable to a number of environments 
and LA tools. 

Recommendations 

This section includes two categories of recommendations. First, recommendations for future 
research are presented. Second, recommendations for professional practice in relation to the 
implementation model are presented. 

Future research. This study could be expanded to include external validation of the model 
presented here. Using the instructor LA implementation model from this study, researchers can 
work with an institution of higher education to study the impact of the model’s use. This type of 
study would also measure the model’s usability (usefulness, effectiveness, efficiency, and 
learnability) in a setting external to the one where the model was developed. 

The model could also be studied in relation to student retention. Researchers could seek to 
study how instructors’ use of this model to implement LA in their courses might affect course 
grades and student persistence. Researchers could also study the effect of the model’s use on the 
overall teaching and learning process. 

Recommendations for practice. The first recommendation is that instructors at SWOSU 
wanting to implement available LA tools and technologies (i.e. Dropout Detective and/or Canvas 
Analytics) use the model presented here to support their efforts. Review of this model will enable 
instructors to better understand how to effectively implement LA in their courses. The model 
demonstrates the benefits of LA and practical and conceptual guidelines to guide LA 
implementation. It also includes some areas of caution that instructors should be aware of so as 
not to fall into common pitfalls in the implementation of LA. The model should be made available 
to SWOSU instructors through the Center for Excellence in Teaching and Learning, and a 
workshop should be offered by the researcher for interested faculty. 

Second, since the model was designed to be generalizable to a number environments, 
instructors at other universities can use the model to implement LA in their course. This model is 
meant to be something that can be adopted and used by individual instructors in individual courses. 
The institution as a whole does not have to implement this model as a standard of practice. 
Instructors can use this model at will, and it should be used only by those who have an interest and 
desire to do so. The researcher will make this model available to any interested parties who might 
put it to use in order to improve their teaching practices. 
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